fix(cli): score a throwing generator as unscorable, so os lint --eval's meanScore stops reading 100 when nothing was generated - #15659

📓 Docs Drift Check

3 anchor(s) derived from 1 changed package(s); no hand-written page names any of them, so this run has nothing to list — not a clean bill of health. This check sees only pages that NAME a derived anchor: one that documents this change in prose, or enumerates it in an authoring dialect, names none and stays invisible to it on every run.

# while this PR is open — GitHub drops the merge commit once it closes
git fetch origin 6a34e7cec181677a771c8454f8401b14e1d0fa3c && git checkout 6a34e7cec181677a771c8454f8401b14e1d0fa3c
# afterwards, rebuild it from the two parents, which stay fetchable
git fetch origin 8c84790201ed07807145a3976871e89e84d64753 dac629d195b8f66ae2eeb4e5b3a083b9b353a160 && git checkout -B drift-repro 8c84790201ed07807145a3976871e89e84d64753 && git merge --no-ff dac629d195b8f66ae2eeb4e5b3a083b9b353a160

node scripts/docs-audit/affected-docs.mjs --json 8c84790201ed07807145a3976871e89e84d64753

Seat rulings on all three open questions

domain:cli execution seat (#6024), session session_01D47qPfEWVPmhguWgBZCi5N.

Q1 — Clause ②: A. The overturn is ACCEPTED. Clause ② is YES.

My provisional NO was incomplete, not merely unlucky. I graded the mechanical floor only — "changes the value of an existing key, adds no key" — and stopped there. You measured that half and confirmed it (25 payload key paths before, 25 after, 0 added, 0 removed), then correctly went on to the limb I never reached: the change re-selects an input class between two already-published verdicts (100 / A / valid: true and 0 / F / valid: false) on a shipped command's --json face.

The doctrine for the non-mechanizable limb is grade yes when the call is not clear, and the asymmetry is the whole reason it exists: a false yes self-corrects in one review round; a false no ships. Three independent priors also point the same way — the card's filing seat, triage, and you — all predicting clause ② before any diff existed.

⭐ What makes the overturn credible is its structure: you did not inflate the mechanical finding to justify the grade. You reported the floor as untriggered, then argued the conformance limb on its own footing. An argument that concedes the half that does not help it is worth far more than one that doesn't.

Your evidence for B is preserved verbatim in the PR body and the clause-② comment, so restoring NO stays cheap if the reviewer disagrees. ⛔ 不预挂 was respected — the label went on card and PR together, in the same round, with the diff in existence, with contrastive read-backs on both carriers.

⇒ The PR parks for at-tier contract review, per the maintainer's ruling 「fable 额度耗尽, pr 应该等契约复审」. That review is dispatched now.

Q2 — meanScore denominator: A. Confirmed. Mean over cases ATTEMPTED.

Your argument is decisive and I am not going to pretend it was a close call:

B alone does not even repair this card: the value the card names as wrong is the score stamped on the FAILED case, which survives untouched if you merely drop that case from a mean.

That is exactly right. Dropping failures from a denominator leaves results[i].score still reading 100 / A / valid: true for a case that produced nothing — the per-case lie the card is actually about. A fixes the per-case value and the mean together; B fixes neither properly.

You also measured the consumers before choosing rather than reasoning about them — exactly two in-repo (the human summary line in commands/lint.ts and the --json payload itself), neither filtering nor re-deriving, so nothing depended on the old denominator. And writing the reason into the payload's own doc comment is what stops the next person from silently switching it.

⇒ A scorable-only statistic remains available as its own card if anyone wants it. It is not a correction to this one.

Q3 — The process conflict: A. You were right, and the contradiction is MINE.

⭐ This is a defect in my dispatch brief, not an ambiguity you had to navigate. My Zone 1 clause 8 said "assign yourself"; the standing os-dev contract says the assignee field belongs to the dispatching PM seat and that the branch, not the assignee, is the identity bit under a shared account. The standing contract is the unconditional clause and wins on conflict. You followed it, and you reported the conflict instead of silently resolving it — which is the only reason I now know my template is wrong.

⇒ I am striking "assign yourself" from this seat's dispatch template. It has gone out in several briefs tonight and will not go out again.

⭐ Worth noting for the record: this exact ambiguity is already being closed in-flight by PR #15290 — "a PM-dispatched card arrives already assigned — claim comments, not the assignee field, say who owns it (#15287)". Your reading is the one the repo is codifying.

The PM-side half-state on the card is mine to close, and I am closing it.

On the report itself

Two things I want on the record because they are the discipline working, not decoration:

• The VOID you declared. The first ablation attempt returned the lock's exit 99 (queue-timeout) and you recorded it as NOT MEASURED — neither pass nor red — with the trap restoring the tree for real, then re-ran on a kept slot. A timeout silently rolled into "we ran it" is how a suite stops meaning anything.
• The narrowing you declared rather than hid. pnpm test for packages/cli was not run (185 files / 2115 tests / 793s against a ~10-minute ceiling on a shared box), and instead of asserting it was fine you measured the blast radius: the only runtime importer is src/commands/lint.ts (driven end to end), corpus.ts imports the MetadataEvalCase type only with its shape unchanged, score.ts mentions it in a comment, and no other test names runMetadataEval / lint --eval / meanScore. That is a narrowing with evidence attached, which is the only kind that counts.

Also noted: check-clause2-carriers.mjs --pair 15659 answered exit 3 PREREQUISITE NOT MET (0 pairs read, repo-scoped REST 403 from that container) and you recorded it as NOT MEASURED rather than as a pass, offering the two contrastive label read-backs as what you could actually attest to. Correct.

#15658 is filed unassigned and bare, with its reachability stated as the question a triager should settle first, and deduped against a search with a live positive control — so the empty result is a reading, not a silent zero. ⛔ Not graded by me.

Generated by Claude Code

ACCEPT WITH FINDINGS — nothing blocking; three non-blocking findings (§6) and two maintainer decisions named, not made (§8).

Contract review at the contract-review tier, commissioned by the domain:cli execution seat (#6024). Reviewed head dac629d195b8f66ae2eeb4e5b3a083b9b353a160 (branch head re-read at 2026-09-05T02:15:12Z, unmoved). Everything below was re-measured in a dedicated detached worktree at that head, with the CLI dependency closure built there; nothing is carried forward from the PR body, the dev report, or the seat's ruling.

VERDICT: PASS
REVIEWED-HEAD: dac629d195b8f66ae2eeb4e5b3a083b9b353a160 (PR #15659)
Implemented-by: `claude/issue-15578-unscorable-throwing-generator`
Reviewed-by: `session_01D47qPfEWVPmhguWgBZCi5N`
FINDINGS: 0 blocking · 3 non-blocking (§6)

Independence, stated rather than implied: the dev was a mode:subagent of the same session as the dispatching seat, so per the 2026-09-02 reading its identity is its branch; this verdict was produced by a context-isolated review subagent of that session. Inputs: the card, the PR (body, diff, comments — including ruling 5548600201), and the repo at head.

1. CI — read at 2026-09-05T02:15:12Z on head dac629d195b

36 check runs (listing complete: 36 of 36 returned). 29 success · 6 skipped · 1 still in progress.

• In progress: Test Core (1/6) (started 01:55:03Z; ~20 min running at read time). ⛔ This reading expires; the landing pre-check must re-read it.
• Lint & Repo Gates: success, completed 02:14:47Z (was in progress at my first read, 02:02:39Z).
• Check Changeset ran twice (runs 33937562923 and 33937625836), both success. Governed Surface Queue Guard: success.
• Skipped (opt-in / not applicable): Packed-tarball smoke ×2, Auto Label and Check PR Size on the second run, Console Pin Gate, Build Docs.

2. Clause ② — independent answer: YES. The overturn stands on its merits, not only by the "when unsure" default.

Doctrine read at head: .claude/skills/pm-dispatch/references/contract-review.md → 载体纪律 → the 机械地板 paragraph, and SKILL.md's Clause-② criterion line ("does this card change accept/reject behaviour or widen the public surface?"). The floor: a new exported symbol or a new key on a published payload is always yes. The conformance class is declared non-mechanizable with two named examples — filling a declared field, and re-selecting an input class between two already-published codes — and grades yes when the call is not clear.

Direction NO, tested first. Mechanical floor: not triggered — re-measured, not inherited. Payload key paths on the bundled 5-case corpus with a throwing generator: 25 before, 25 after, identical sorted set (array indices normalised). No exported symbol: metadata-eval.ts is not on @objectstack/cli's exports (. and ./console only) and unscorableScore is module-private. Accept set: every --generator module is accepted exactly as before. ok / passed / failed / total / exit code: identical before and after (§3). On the one-line criterion alone the answer is NO, and the seat's provisional reading was correct as far as it went.

Direction YES, and why it wins. results[i].score.grade is a closed five-member vocabulary and results[i].score.valid a two-member one, both on a shipped command's --json face. For one input class — a generator that throws — the payload moves from one already-published member (A / valid: true, the empty-stack verdict) to another already-published member (F / valid: false, the sentinel #15576 published for the unwalkable class). A consumer that keys on score.valid or score.grade rather than passed gets a different answer for the same input on the same command. That is the doctrine's second named example almost verbatim. The seat graded the floor and stopped; the dev conceded the floor and argued the limb; the limb reaches.

What the published contract now promises that it did not before (each measured end to end, §3):

1. generationError present ⇒ score is the unscorable sentinel { score: 0, grade: "F", valid: false, counts all 0, schemaErrors: [], issues: [] } — for both causes, thrown and unwalkable. Before: only the unwalkable cause; a thrown generator carried the empty-stack verdict 100 / A / valid: true.
2. generationError present ⇒ score.valid === false. Before, the face asserted valid: true ("schema-valid AND zero lint errors") about a stack that was never produced.
3. meanScore is a mean over every case attempted; a failed generation contributes 0 and is counted. Consequence a consumer may now rely on: meanScore === 100 entails every case produced a scorable stack; before it was compatible with zero stacks produced.
4. Unchanged, and now pinned in both legs: passed, ok, failed, total, exit code, the 25-path key set, the --generator accept set.

The human face moves the same way (100/100 (A) → 0/100 (F) per failed case; mean 100/100 → mean 0/100), driven without --json.

3. The behaviour change, re-driven

• scoreMetadata({}) from source through tsx: {"score":100,"grade":"A","valid":true,"counts":{"schemaErrors":0,"errors":0,"warnings":0,"suggestions":0},"schemaErrors":[],"issuesLen":0}. The card's premise holds.
• End to end through packages/cli/bin/run-dev.js (source entry), lint --eval --json --generator throws.mjs, exit codes captured after redirection, never through a pipe:
  • BEFORE (merge-base blob 7986fe93ee swapped in under a trap): exit 1 · ok:false · passed:0 · failed:5 · meanScore:100; every case 100 / A / valid:true / passed:false / generationError:"model unavailable".
  • AFTER (head blob f7b4ac2a1bb): exit 1 · ok:false · passed:0 · failed:5 · meanScore:0; every case 0 / F / valid:false / passed:false, same generationError.
  • stderr 0 bytes on both legs. Restore verified by blob-hash equality with HEAD and a 0-byte git diff HEAD. No build step between legs — the behaviour moved with the source blob alone, which is the proof the measured path is src/.
• Both test legs at head: Test Files 2 passed (2) · Tests 29 passed (29), exit 0.
• passed / ok / failed genuinely unchanged: passed: !generationError && … is untouched by the diff, and the BEFORE/AFTER pair shows identical flags and exit code. A repair that had moved passed would have shown up here; it did not.

4. Denominator — A (mean over cases attempted): written, and pinned by a test that fails on a silent switch

• Doc comment: present at head on MetadataEvalReport.meanScore (denominator is every case attempted; why scorable-only is a different metric; total / passed / failed carry the counts) and on MetadataEvalCaseResult.score.
• Pin: the failed case is COUNTED in the denominator, not dropped from it. I ablated the reducer to scorable-only (results.filter((r) => !r.generationError)), predicting exactly one red in the unit file. Measured: Tests 1 failed | 15 passed (16) — the predicted test, AssertionError: expected 100 to be 50. Mutation proven on disk (marker 0→1, blob f7b4ac2a1bb → dd2211003a9); restore proven (blob back to f7b4ac2a1bb, marker 0, 0-byte diff, clean status).
• The pin is non-degenerate: the clean fixture scores 100, so the two denominators answer 50 vs 100. (Had it scored 1, Math.round(1/2) = 1 would collide with the scorable-only value — noted only so nobody later "simplifies" that fixture.)
• The seat's reasoning holds: route B alone leaves results[i].score at 100 / A / valid: true on the failed case, which is the value the card names.

5. Gates, read at source

• Bump — patch, consistent with the rule as written. WHICH LEVEL read in .github/workflows/pr-automation.yml → Check Changeset job → "Require a changeset (or the skip-changeset label)" step, cross-referenced from scripts/check-changeset-no-major.mjs's header ("CHOOSING BETWEEN THE TWO LEVELS THIS GUARD LEAVES"). Its only at-least-minor trigger is a purely additive widening — a new exported symbol on an index, a new accepted key or value. None is present (§2). Closest precedent: sibling fix(cli): os lint --eval --json reports an unscorable stack instead of crashing #15576, same file, same payload, patch. See finding 6.2.
• ADR-0087 — not engaged. Measured against breakingDeclaration()'s three signals in scripts/check-adr-0087-registration.mjs: bump patch (not major); no **BREAKING / BREAKING CHANGE in the body; first body line is not a type!: summary. No marker required, none present. Correct.
• ADR-0112 — N/A. Added lines contain no toThrow and no error-code assertion.
• Governed surfaces — none. The PR surface from merge-base d30ccb9bd96 is exactly 4 files: the changeset, packages/cli/src/lint/metadata-eval.ts, and the two test files. ⚠️ A two-dot diff against the API's recorded base.sha 8c84790201e sweeps in 42 files of main's own drift, including skills/objectstack-platform/SKILL.md — not this PR's. Anyone re-checking must diff from the merge-base.
• Declared narrowing (pnpm test for packages/cli not run) — population claim verified on the head tree; the narrowing stands. Every file naming metadata-eval / runMetadataEval / meanScore / MetadataEvalCase / DEFAULT_METADATA_EVAL_CORPUS / generationError / lint --eval, repo-wide excluding node_modules and dist: runtime importer src/commands/lint.ts only; src/lint/corpus.ts imports the MetadataEvalCase type only (interface untouched by the diff); src/lint/score.ts a comment; the two test files in the PR. meanScore occurs in exactly three files (module, lint.ts, unit test). Prose-only mentions: skills/objectstack-data/SKILL.md (pass bar; no meanScore semantics), docs/qa/platform-checklist/areas/cli.json (explicitly out of that item's scope), packages/cli/CHANGELOG.md and the sibling changeset (historic, about the conversions key's absence), one unrelated docs/audits line. No gate script or fixture JSON carries the payload; content/docs has zero mentions of lint --eval, meanScore, --eval-min, --generator. Sibling objectui (at a472b07, read-only): 0 hits, with a positive control. Sibling cloud: NOT MEASURED (not present in this session). CI's own Test Core shards: 5 of 6 green, (1/6) still running (§1).

6. Findings — all non-blocking

1. The eval --json payload has no user-facing documentation. "Written into the payload's own documentation" is true in exactly one sense: the TS docblock on an interface in a module that is not on the package's exports — which is the only documentation the payload has. Not a defect of this PR; a maintainer product question (§8).
2. WHICH LEVEL is silent on this class. The rule names additive widening only; "an already-published value changes meaning for one input class, with no new key" is addressed neither way. patch is right under the rule as written and under the fix(cli): os lint --eval --json reports an unscorable stack instead of crashing #15576 precedent; clause ② is the channel that routes this class to a human, and it did. Whether the prose should say so is the maintainer's call, not this PR's.
3. The changeset describes the --json face; the human face moves too (N/100 (A) → 0/100 (F), and the mean line). The headline sentence covers both since the mean line prints meanScore; one clause naming the human face would remove the ambiguity. Cosmetic.

7. The dev's open questions, disposed

• Q1 clause ②: YES — agreed, on merits (§2).
• Q2 denominator: A — agreed; written and pinned (§4). A scorable-only statistic would be a new key and therefore a mechanical-floor yes on its own card, if the maintainer ever wants it.
• Q3 assign-yourself conflict: not a contract matter; the seat has closed it.

8. Maintainer decisions named, not made

• Whether os lint --eval --json gets a user-facing page (currently none — its only description is the interface docblock).
• Whether WHICH LEVEL should address value-meaning changes on a published payload.

Nothing here flips ready, enqueues, arms auto-merge, or touches labels; the landing pre-check — including a fresh read of Test Core (1/6) — is the seat's. #15658 was not graded here.

Generated by Claude Code

Generated by Claude Code